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(54) Image retrieval apparatus and method 

(57) An image feature amount extraction unit seg- 
nnents an image into a plurality of blocks and calculates 
the feature amount of each block. A feature amount la- 
bel string generation unit adds a label to each block in 
accordance with the feature amount acquired for the 
block and arranges the labels on the basis of a prede- 
termined block sequence to generate a label string. In 
retrieval, the similarity between the label string of a 



specified image and that of an image to be compared is 
calculated, and an image whose similarity exceeds a 
predetermined value Is output as a retrieval result. In 
this way. similar image retrieval can be performed in 
consideration of the arrangement of feature amounts of 
images, and simultaneously, similar image retrieval can 
be performed while absorbing any difference due to a 
variation in photographing condition or the like. 
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Description 

The present invention relates to an image retrieval 
apparatus and method of retrieving an image. 

Conventionally, various techniques of retrieving an 
image have been proposed. Many similar image retriev- 
al techniques tor natural images, which are put into prac- 
tice to some extent, use color information as an image 
feature amount. Most of these techniques calculate the 
histogram associated with color information and retrieve 
an image using the RGB ratio or combination of majority 
colors in images. 

In these techniques, however, the retrieval accura- 
cies are not always high because positional information 
of colors is lost. For example, Japanese Patent Laid- 
open No. 8-249349 discloses pattern matching using 
feature amounts (representative colors) of a plurality of 
blocks obtained by segmenting each image. With this 
technique, however, the distance in feature amount be- 
tween blocks of two images to be matched must be cal- 
culated, so an enormous amount of calculation is re- 
quired. Especially, when representative colors are used 
as feature amounts, three data, i.e., R, G, and B data 
must be handled, and calculation becomes more com- 
plex. In addition, since the feature amounts themselves 
are used to compare images, similar image retrieval is 
disabled only when the image angle or object position 
changes although the comparison accuracy rises. More 
specifically, appropriate image retrieval while ensuring 
some degree of ambiguity by absorbing some difference 
in image feature amount due to a change in image angle 
or object position or photographing conditions, i.e., so- 
called robust similar Image retrieval cannot be per- 
formed. 

When a natural image is to be retrieved using the 
conventional technique, normally, keywords are added 
to images in advance, and image retrieval is performed 
on the basis of the keywords. However, keyword addi- 
tion is labor-intensive. Additionally, an image having no 
keyword must be manually selected by displaying 
thumbnail images, resulting in cumbersome retrieval. 

One aspect of the present invention provides an im- 
age retrieval apparatus and method which allows high- 
speed similar image retrieval in consideration of the im- 
age feature amount arrangement. 

Another aspect of the present invention provides an 
image retrieval apparatus and method which allows sim- 
ilar image retrieval by effectively absorbing a change in 
image due to a variation in photographing condition 
such as a photographing angle or photographing mag- 
nification. 

According to an aspect of the present invention, 
there is provided an image retrieval apparatus compris- 
ing addition means for segmenting an image into a plu- 
rality of blocks and adding a label to a feature amount 
acquired in correspondence with each block, generation 
means for arranging labels added by the addition meeuns 
on the basis of a predetermined block sequence to gen- 



erate a label string, storage means for storing the label 
string generated by the generation means in corre- 
spondence with the image, and retrieval means for-per- 
forming image retrieval on the basis of the label string 

s stored in the storage means. 

According to another aspect of the present inven- 
tion, in the image retrieval apparatus, the label string 
represents a two-dimensional label matrix, and the re- 
trieval means comprises first matching means for mak- 

10 ing a label string extracted from a label matrix of a re- 
trieval specified image in units of rows correspond to a 
label string extracted from a label matrix of an image to 
be compared in units of rows by DP matching to obtain 
a row array of the image to be compared, and second 

15 matching means for obtaining, by DP matching, a simi- 
larity between a row array of the retrieval specified im- 
age and the row array obtained by the first matching 
means. 

With this arrangement, two-dimensional DP match- 
20 ing can be performed. 

An aspect of the present invention enables quick 
similar image retrieval by representing a feature amount 
group by one symbol and calculating the similarity be- 
tween labels to reduce the similarity calculation amount. 
2S An aspect of the present invention performs satis- 
factory similar image retrieval by representing similar 
feature amounts by one label. 

Another aspect of the present invention more effec- 
tively performs similar image retrieval by applying a 
30 technique such as DP matching or fuzzy nondetermin- 
istic automaton which allows certain ambiguity in label 
positions. 

Embodiments of the present invention will now be 
described with reference to the accompanying draw- 
as ings, in which: 

Fig. 1 is a block diagram showing the control ar- 
rangement of an image retrieval apparatus accord- 
ing to the first embodiment; 

Fig. 2 is a block diagram showing the functional ar- 
rangement of the image retrieval apparatus accord- 
ing to the first embodiment; 
Fig. 3 is a view for explaining the image data storage 
state in an image storage-unit 17; 
Fig. 4 is a flow chart showing the procedure of im- 
age registration processing according to the first 
embodiment; 

Fig. 5 is a view showing an example of image block 
division according to the first embodiment; 
Fig. 6 is a view for explaining a multidimensional 
feature amount space according to the first embod- 
iment; 

Figs. 7A to 7C are views for explaining block se- 
quences in generating a label string; 
Fig. & is a view showing an example of the data 
structure of an image management DB record; 
Fig. 9 is a view showing an example of the data 
structure of a label system index; 



40 



45 



so 



ss 



3 EP 0 866 409 A1 4 



disk, a floppy disk, or a CD-ROM. A network interface 
108 allows communication with devices on a network. 
Reference numeral 109 denotes an interface; and 110, 
a scanner for reading an image. A bus 111 connects the 

s above-described elements. The program for realizing 
processing shown in flow charts (to be described later) 
may be stored in the ROf^ 102 or loaded from the ex- 
ternal storage unit 106 to the RAM 103. 

In the above arrangement, the scanner 110 or ex- 

10 temal storage unit 1 06 may be substituted with a device 
on the network. 

Fig. 2 is a block diagram showing the functional ar- 
rangement of the image retrieval apparatus of the first 
embodiment Referring to Fig. 2, a user interface unit 11 

15 detects various operation inputs from the user using a 
display device 107, the keyboard 104, and the mouse 
105- An image input unit 12 reads an image with the 
scanner 110. An image memory 13 stores image data 
obtained from the image input unit 1 2 in a predetemiined 

20 area of the RAM 103. An image feature amount extrac- 
tion unit 14 extracts the feature amount of the image 
stored in the image memory 1 3 following a procedure to 
be described later. A feature amount label string gener- 
ation unit 15 generates a label string on the basis of the 

25 feature amount obtained by the image feature amount 
extraction unit 14. A pattern matching unit 16 calculates 
the similarity between a user specified image and imag- 
es stored in an image storage unit 17 on the basis of 
their label strings to search and retrieve a similar image. 

30 The image storage unit 17 stores Image data ob- 
tained from the image input unit 12 or the like. Fig. 3 is 
a view for explaining the image data storage state in the 
image storage unit 17. An image ID 111 is added to each 
Image data 112, so one image ID 111 and one image 

35 data 112 are paired and held in the image storage unit 
17. An image management database (to be referred to 
as an image management DB hereinafter) 1 8 manages 
Image data stored in the image storage unit 17 in the 
data format shown in Fig. 8. A label string Index 19 

40 stores a label system index file shown in Fig. 9 or label 
component index file shown in Fig. 11 . Use of these in- 
dices will be described later with reference to the flow 
chart In Fig. 10, 

The operation of the image retrieval apparatus ac- 

45 cording to the first embodiment with the above arrange- 
ment will be described below. The following description 
will be made by exemplifying processing in a three-di- 
mensional color space while employing colors, three 
colors of red (R), green (G), and blue (B). in this case, 
50 as image feature amounts. 



Fig. 10 is a flow chart for explaining the procedure 
of similar image retrieval according to the first em- 
bodiment; 

Fig. 11 is a view showing an example of the data 
structure of a label component Index according to 
the first embodiment; 

Fig. 1 2 is a view showing a penalty matrix between 
labels used to compare label strings to obtain sim- 
ilarity; 

Figs. 1 3A to 1 3C are views for explaining calcula- 
tion of the distance between label strings by DP 
matching; 

Fig. 1 4 Is a block diagram showing the functional 
arrangement of an image retrieval apparatus ac- 
cording to the second embodiment; 
Fig. 1 5 is a flow chart showing the procedure of im- 
age registration processing according to the second 
embodiment; 

Figs. 1 6A to 1 6E are views for explaining block se- 
quences in generating a label matrix in the second 
embodiment; 

Fig. 17 Is a view showing an example of the data 
structure of an image management DB record ac- 
cording to the second embodiment; 
Fig. 18 is a view showing an example of the data 
structure of a label system index according to the 
second embodiment; 

Fig. 19 is a flow chart for explaining the procedure 
of similar image retrieval according to the second 
embodiment; 

Fig. 20 is a view showing an example of the data 
structure of a label component index according to 
the second embodiment; 

Figs. 21 A to 21 C are views for explaining similarity 
calculation according to the second embodiment; 
Fig. 22 is a flow chart for explaining the procedure 
of similarity calculation employing DP matching in 
the second embodiment; 

Fig. 23 is a flow chart showing the procedure of dy- 
namic penalty value setting according to the second 
embodiment; and 

Fig. 24 is a view for explaining adjustment of width 
of matching window. 

[First Embodiment] 

Fig. 1 is a block diagram showing the control ar- 
rangement of an image retrieval apparatus according to 
the first embodiment. Referring to Fig. 1 , a CPU 1 01 ex- 
ecutes various control operations in the image retrieval 
apparatus of this embodiment. A ROM 102 stores a 
bootstrap program to be executed in starting the appa- 
ratus and various data. A RAM 103 stores a control pro- 
gram to be executed by the CPU 1 01 and also provides 
a work area for the CPU 1 01 to execute various control 
operations. A keyboard 104 and a mouse 105 provide 
various environments for user inputs. 

An external storage unit 1 05 is constituted by a hard 



[Image Registration Processing] 

Image registration processing will be described 
55 first. Fig. 4 is a flow chart showing the procedure of im- 
age registration processing according to the first em- 
bodiment. In step S11 . an image is loaded using the im- 
age input unit 12 in accordance with an instruction from 



3 



SDOCID:<EP 0B66409A1 l.> 



5 



EP 0 866 409 A1 



6 



the user through the user interface unit 11 and held in 
the Image memory 13. In step SI 2. this image is seg- 
mented Into a plurality of blocks. In this embodiment, the 
image is broken up into a plurality of blocks in the vertical 
and horizontal directions. Fig. 5 is a view showing an 
example of image block division according to this em- 
bodiment. As shown In Fig. 5, in this embodiment, the 
image is segmented into 3x3 blocks, i.e., a total of 9 
blocks for the descriptive convenience. In step SI 3, the 
feature amounts of the segmented blocks are calculat- 
ed, so labels of the obtained feature amounts are gen- 
erated according to the following procedure. 

The division into 3X3 blocks in this embodiment 
is merely an example for the descriptive convenience. 
Actually, a natural image is preferably segmented into 
10 X 10 or more blocks. For example, an image of a 
product with a white background is preferably segment- 
ed into 1 3 X 13 or more blocks. 

Fig. 6 is a view for explaining a multidimensional 
feature amount space according to this embodiment. 
The multidimensional feature amount space (RGB color 
space) is segmented into a plurality of blocks (color 
blocks), i.e.. cells (color cells), and unique labels with 
serial numbers are added to the respective cells (color 
cells). The multidimensional feature amount space 
(RGB color space) is segnnented into a plurality of blocks 
to absorb the fine difference in feature amount (color). 

For the multidimensional feature amount space, the 
image feature amount is not directly used. Instead, the 
average and variance of each parameter is standard- 
ized (normalized) by experiments, and then, orthogonal 
transformation such as principal component analysis is 
performed to obtain a significant dimension. Note that 
the "significant dimension" is a dimension defined by a 
principal component axis having a high contribution ratio 
in principal component analysis. 

In step SI 3, each of segmented blocks obtained in 
step SI 2 is subjected to predetermined image feature 
amount calculation to determine a cell on the multidi- 
mensional feature amount space, to which the block be- 
longs, and obtain a corresponding label. All blocks are 
subjected to this processing. More specifically, each 
segmented image block is subjected to calculation for 
obtaining color cells to which pixels of the image block 
belong. The label of a color cell with the highest frequen- 
cy is determined as the parameter label (color label) of 
the segmented image block. This processing is per- 
formed for all blocks. 

The parameter labels are added to the blocks in the 
above-described manner. In step SI 4, the parameter la- 
bels added to the blocks are arranged on the basis of a 
predetermined block sequence to generate a parameter 
label string (to be referred to as a label string hereinaf- 
ter). Figs. 7A to 7C are views for explaining block se- 
quences in generating a label string. The parameter la- 
bels are arranged in accordance with numbers in the 
segmented inrrage blocks to generate a label string. 
In Fig. 7 A, the segmented blocks are obliquely 



scanned from the upper right side to the bwer left side. 
This is because the influence of any fine difference or 
shift in angle between images to be compared is mini- 
mized by obtaining continuous label strings as many as 

s possible with high expectation along the object to be 
subjected to similarity retrieval. Consequently, in coop- 
eration with the pattern matching unit 16 (to be de- 
scribed later), strings can be compared with each other 
while being less affected by a shift in both the vertical 

10 and horizontal directions. 

The following scanning methods can be applied to 
this embodiment. 

Horizontal scanning (four scanning methods are 
available; e.g., scanning from the left side to the right 

IS side is performed from the upper side to the lower side, 
or scanning from the left side to the right side is per- 
formed from the lower side to the upper side). 

Vertical scanning (four scanning methods are avail- 
able; e.g., scanningfromthe upper side tothe lower side 

20 is performed from the left side to the right side). 

Oblique scanning (oblique scanning methods in two 
directions are available for each of starting points at the 
four corners, so a total of eight scanning methods are 
available including those shown in Figs. 7A to 7G). 

25 Zigzag scanning (zigzag scanning is employed in, 
e.g., JPEG. Two zigzag scanning methods are available 
for each of starting points at four corners, so a total of 
eight scanning methods are available). 

In this embodiment, the scanning method to be em- 

30 ployed is determined from the following viewpoints. 

(1) In this embodiment, label strings are subjected 
to time series comparison, so reversal of this se- 
quence is not preferable. All images must be 

35 scanned by a predetermined scanning method to 
generate label strings. 

(2) Blocks close to each other are preferably ar- 
ranged close even in a label string. 

(3) When the labels of blocks associated with the 
40 object to be retrieved can be detected as soon as 

possible and continuously scanned as long as pos- 
sible, matching is facilitated. 

(4) Even when the object is moved or changed in 
angle, the label array must be prevented from being 

45 excessively changed. 

The scanning method to be employed need satisfy the 
above conditions. Especially, assuming that many ob- 
jects of interest are located at the center of images, this 

50 embodiment employs oblique scanning as a method 
with high expectation for detecting blocks including the 
object of interest as soon as possible and scanning the 
object of interest as long as possible. In this embodi- 
ment, oblique scanning from the upper right side to the 

55 lower left side as shown in Fig. 7A is employed. How- 
ever, scanning as shown in Fig. 7B or 7C may be em- 
ployed, as a matter of course. 

In step SI 5, the label string or image data obtained 
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in the above manner is stored in the image storage unit 
(^"' 17. the image management DB 18, and the label string 
index 1 9. More specifically, an image ID Is acquired for 
the Image data loaded In step S11. and the Image ID 
and image data are paired and stored in the image stor- 
age unit 17. An image management DB record shown 
in Fig. 8 is generated in correspondence with the image 
ID and registered In the image management DB 18. In 
step SI 6. the Image ID Is registered in the label string 
index 19 storing a record (label system index shown in 
Fig. 9) in which an image ID group is stored in a variable 
length record while setting label strings as retrieval keys. 
If the label string has not been registered, a new record 
is generated, and a label string ID Is added to the label 
string, so that the label string and corresponding image 
ID are registered. If the label string has already been 
registered, the innage ID is additionally registered in the 
image ID group. By using such a label system index, an 
image ID corresponding to a given label string can be 
quickly obtained. 

Image registration processing has been described 
above. 

[Similar Image Retrieval Processing] 

Similar image retrieval processing will be described 
next with reference to the flow chart in Fig. 10. Fig. 10 
is a flow chart for explaining the procedure of similar im- 
age retrieval processing. In this embodiment, the label 
string group of already registered images is obtained 
from the label system index at the time of initialization 
in advance, and a label component index file (Fig. 11) 
having label components as keys is generated and 
stored in the label string index 19. In this case, the time 
of initialization means the time o1 starting the system or 
the time of starting an application. Even when a new im- 
age is registered in the image DB, the label component 
index is generated. Fig. 11 is a view showing the data 
structure of the label component index. As shown in Fig. 
1 1 , each label component of the label component index 
has an address group (string ID group) corresponding 
to the label strings having the labels. The label compo- 
nent index file need not be regenerated until image reg- 
istration, deletion, or change need be reflected on the 
label component index. Note that, the label component 
index may be updated or added when a new image is 
registered. 

In step 321, a similar image retrieval specified im- 
age is designated from the user interface unit 1 1 . In step 
S22, the Image ID of the designated similarity retrieval 
specified image (to be referred to as a specified image 
hereinafter) is acquired, and the label string (color label 
string in this example) of the specified image is acquired 
from the image management DB 18. 

In step S23, a label string group (label strings in the 
label system index) including a predetermined number 
or more labels similar to the label string of the specified 
image is acquired with reference to the label component 



index file. If the label string of the specified Image is com- 
pared with all label strings of registered images, the 
processing speed lowers. To prevent this, similar label 
strings (label string group including a predetermined 

s nunnber or more labels having small penalties to the la- 
bel string of the specified image, the penalty will be dis- 
cussed later referring to Fig. 12) are extracted in ad- 
vance, and then, each of these label strings is compared 
with the label string of the specified image, thereby im- 

10 proving the processing speed. If a low processing speed 
is permitted, the label string of the specified image may 
be compared with all label strings of registered images 
to perform highly accurate retrieval (in this case, step 
S23 is omitted). 

is The following method is available as a method of 
extracting images using the above-described label com- 
ponent index file. In this embodiment, a label string in- 
cluding nine labels is used, and images including three 
labels (fourth to sixth labels) at the center of this label 

20 string are extracted. This is because, generally, the fea- 
ture information of the image is present at the central 
portion of the image. 

In step S24, each of the label strings acquired in 
step S23 is compared to the label string of the specified 

2S image to calculate similarity. The label strings are se- 
quentially output from the closest one to the label string 
of the specified image together with their similarities as 
a detection result. 

The method of performing similarity comparison be- 

30 tween label strings (calculation of similarity) will be de- 
scribed. 

Fig. 1 2 is a view showing a penalty matrix between 
labels used in label string comparison for obtaining the 
similarity. A smaller value in the matrix indicates higher 

35 similarity. For example^ the penalty between label 2 and 
labels is "7". The penalty between identical labels is "0", 
as a matter of course. This matrix Is used to perform 
distance determination corresponding to the similarity 
between labels. More specifically, in this embodiment, 

40 since the RGB color space is used as the feature 
amount space, distance determination corresponding to 
the similarity between colors can be made. 

For example, the distance between label values at 
corresponding positions in the label strings of the re- 

45 trieval specified image and the image to be compared 
is obtained with reference to the penalty matrix shown 
in Fig. 12, and the sum of distances for all labels in the 
label strings is calculated, thereby obtaining the dis- 
tance between the label strings. In the example shown 

50 in Fig. 1 3A, the label string of the specified image is 
■112313441", and the label string of the image to be 
compared is "113224452". The distance (final solution) 
is obtained by performing DP matching using the pen- 
alty matrix shown in Fig. 12, as shown in Fig. 13B. In 

55 this example, the following condition is used as a gradi- 
ent constraint. In Fig. 13C, when the costs on the lattice 
points (i-IJ). (i-1.H). and (i,j-1) are g(i-1 J), g(i-1.i-1). 
and g(i,j-1), respectively, and the penalty on the lattice 
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point (i.j) is d(i,j), the cost g(i,j) on the lattice point (i.j) is 
obtained by the following recurrence formula: 



g(i,j) = nun 



gaj-1) + dOJ) 
gO'Lj'l) + 2d(i,j) 
g(i-lj) d(Uj) 



To give a small penalty (distance) to cells adjacent 
to each other or a large penalty to cells spaced apart 
from each other in pattern matching between labels^ the 
penalty matrix between labels as shown in Fig. 12 is 
used. In step S24, the label strings are compared with 
each other in consideration of the penalty matrix. At this 
time, matching such as automaton which allows ambig- 
uous comparison between label sequences may be per- 
formed. When such an ambiguous comparison method 
is used, a small penalty is given to excess label addition, 
label omission, or repeat of a label. In addition, when 
the distance between label strings is calculated using 
the penalty matrix between color labels shown in Fig. 
12, ambiguous pattern matching can be performed. The 
"fuzzy nondeterministic finite automaton" described in 
"ambiguous character string retrieval method and sys- 
tem using fuzzy nondeterministic finite automaton" dis- 
closed in Japanese Patent Laid-Open No. 8-241 335 can 
be applied as the automaton. In this automaton, a mul- 
tivalued distance (penalty) between symbols can be ob- 
tained. As a method which allows to ambiguously move 
the positions of labels to be compared to realize label 
string comparison such that the total distance is mini- 
mized (similarity is maximized), not only the above-de- 
scribed automaton but also DP matching used in speech 
recognition may be used. This method can also be pref- 
erably applied to this embodiment. 

When the rule of block sequences in Figs. 7A to 7C 
is employed in addition to the above-described ambig- 
uous pattern matching, color label strings can be com- 
pared with each other while being less affected by any 
fine difference or shift in angle between the images to 
be compared and also less affected by a shift in both 
the vertical and horizontal directions. More specifically, 
DP matching or fuzzy nondeterministic automaton per- 
mits certain ambiguity in sequence of the label strings 
and absorbs the influence of a positional shift in image. 
When the position of the object changes due to the dif- 
ference in angle, and the position of the object to be ex- 
tracted changes depending on the block, the hue of the 
block is expected to finely change. This difference is ab- 
sorbed by the above-described penalty matrix. As de- 
scribed abovC; by the synergy of DP matching or match- 
ing based on fuzzy automaton allowing ambiguity and 
the allowance of ambiguity in feature amounts by the 
penalty matrix, matching less affected by a shift in the 
vertical and horizontal direction is realized. In addition, 
since the change in label position due to a change in 



object position is minimized by employing oblique scan- 
ning shown in Figs. 7A to 7C, the influence of a shift in 
object in matching can be more effectively minimized. 
In step S25. an Image search is made from the label 

S system index using the label string with high similarity 
as a key, and the image ID of the found image is ac- 
quired. This processing is repeated for label strings out- 
put in descending order of similarities to obtain the im- 
age I D group of similar images. In step S26, the full path 

10 file name for each image ID of the image ID group is 
acquired with reference to the image management DB 
18 and presented to the user. 

With the above processing, some difference in color 
due to a change in image angle, object position, or pho- 

15 tographing condition is absorbed, so robust similar im- 
age retrieval can be performed at a high speed. 

As described above, according to the first embodi- 
ment, the feature amount group (group of feature 
amounts obtained by segmenting the feature amount 

20 space) is represented by one symbol (i.e. , a label is gen- 
erated), and a distance based on the similarity between 
labels is given on the basis of the penalty matrix. With 
this arrangement, the amount of calculation of the dis- 
tance between two image blocks can be largely re- 

2S duced. In addition, since similar feature amounts are 
represented by one label, similar image retrieval can be 
satisfactorily performed. 

When (1) the concept of distance between labels 
based on the penalty matrix is introduced, and (2) a 

30 method such as DP matching or fuzzy nondeterministic 
automaton which allows to ambiguously move the posi- 
tions of labels to be compared to realize label string 
comparison such that the total distance is minimized 
(similarity is maximized) is introduced, similar image re- 

35 trieval can be performed even when the image angle 
slightly changes, so images having similar atmospheres 
can be retrieved. 

Furthermore, the speed of image retrieval can be 
further increased by using the index database (label 

40 system index or label component index). 

As described above, according to the present in- 
vention, similar image retrieval can be performed in con- 
sideration of the arrangement of the feature amounts of 
images. 

45 

[Second Embodiment] 

In the first embodiment, similar image retrieval is 
performed by one-dimensionally arranging labels repre- 
so senting feature amounts, i.e., using a label string. In the 
second embodiment, similar image retrieval is per- 
formed in consideration of two-dimensionally arranged 
labels representing feature amounts, i.e., using a label 
matrix. 

ss The control arrangement of an image retrieval ap- 
paratus according to the second embodiment is the 
same as that of the first embodiment (Fig. 1 ), and a de- 
tailed description thereof will be omitted. 
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Fig. 14 is a block diagram showing the functional 
arrangement of the image retrieval apparatus according 
to the second embodiment. The same reference numer- 
als as in the first embodiment denote the same elements 
in Fig. 14, and a detailed description thereof will be omit- s 
ted. A feature amount label matrix generation unit 15a 
generates a label matrix on the basis of feature amounts 
obtained by an inrage feature amount extraction unit 1 4. 
A pattern matching unit 1 6a calculates the similarity be- 
tween the label matrix of a designated image and that io 
of an image stored in an image storage unit 17 using 
two-dimensional DP matching (to be described later). 
An image management database (to be referred to as 
an image management DB hereinafter) 18a manages 
image data stored in the image storage unit 17 in the i5 
data format shown in Fig. 17. A label matrix index 19a 
stores a label system index shown in Fig. 1 8 or a label 
component index file shown in Fig. 20. 

The operation of the image retrieval apparatus ac- 
cording to the second embodiment with the above ar- 20 
rangement will be described below. The following de- 
scription will be made by exemplifying processing In a 
three-dimensional color space while employing colors, 
three colors of red (R), green (G). and blue (B), in this 
case, as image feature amounts, as in the first embod- 2S 
iment. 

[Image Registration Processing] 

Fig. 15 is a flow chart showing the procedure of im- 30 
age registration processing according to the second em- 
bodiment. Steps S11 to SI 3 in Fig. 15 are the same as 
those In the first embodiment (Fig. 4), and a detailed 
description thereof will be omitted. 

With processing in steps S1 1 to SI 3, parameter la- 35 
bels are added to blocks. In step SI 4a, the parameter 
labels added to the blocks are arranged on the basis of 
a predetermined block sequence to generate a param- 
eter label matrix (to be referred to as a label matrix here- 
inafter). Figs. ISA to 1 6E are views for explaining block 40 
sequences in generating a label matrix. The parameter 
labels are arranged in accordance with numbers in the 
segmented image blocks to generate a label matrix. 
When a label matrix is to be stored in the image man- 
agement DB 18a or label matrix index 1 9a, two-dimen- 
sional label matrixes are one-dimensionally arranged in 
a predetermined order and stored, as described above. 
In the second embodiment, this one-dimensional array 
will also be referred to as a label matrix. 

In Fig. 16A, the segmented blocks are scanned in 50 
the horizontal direction from the left side to the right side, 
and this horizontal scanning is performed from the upper 
side to the lower side. The following scanning methods ■ 
can be applied to this embodiment. 

Horizontal scanning (not only horizontal scanning 5S 
(Fig. 1 6A) in which scanning from the left side to the right 
side is performed from the upper side to the lower side, 
scanning methods shown in Figs. IBBto 16D are also 



available: e.g., scanning from the left side to the right 
side is performed from the lower side to the upper side, 
so a total of four scanning methods are available). 

Vertical scanning (four scanning methods are avail- 
able; e.g., scanningfromthe upper side to the lower side 
is performed from the left side to the right side). 

As shown in Fig. 16E, scanning Is performed in units 
of even or odd lines. 

In this embodiment, the scanning method shown in 
Fig. 16A is employed, although the above-described 
scanning methods can also be applied. 

In step SI5a, the label matrix and image data ob- 
tained in the above manner are stored in the image stor- 
age unit 17, the image management DB 18a, and the 
label matrix index 1 9a. More specifically, an image ID is 
acquired for image data loaded in step S11 , and the im- 
age ID and image data are paired and stored in the im- 
age storage unit 17. An image management DB record 
shown in Fig. 17 is generated in correspondence with 
the image ID and registered in the image management 
DB 18a. In step 81 6a, the label matrix Is registered in 
the label matrix index 19a having a record (label system 
index shown in Fig. 18) in which an image ID group is 
stored in a variable length record while setting the label 
matrixes as retrieval keys. If the label matrix has not 
been registered, a new record is generated, and a label 
matrix ID is added to the label matrix, so that the label 
matrix and corresponding image ID are registered. If the 
label matrix has already been registered, the image ID 
is additionally registered in the image ID group. By using 
such a label system index, an image ID corresponding 
to a given label matrix can be quickly obtained. 

Image registration processing has been described 
above. 

[Similar Image Retrieval Processing] 

Similar image retrieval processing will be described 
next with reference to the flow chart in Fig. 19. Fig. 19 
is a flow chart for explaining the procedure of similar im- 
age retrieval processing. In this embodiment, the label 
matrix group of already registered images is obtained 
from the label system index at the time of initialization 
in advance, and a label component index file having la- 
bel components as keys is generated and stored in the 
label matrix index 19a. In this case, the time of initiali- 
zation means the time of starting the system or the time 
of starting an application. Even when a new image is 
registered in the image DB. the label component index 
Is generated. Fig. 20 Is a view showing the data structure 
of the label component index. As shown in Fig. 20, each 
label component of the label component index has an 
address group (matrix ID group) corresponding to the 
label matrixes having the labels. The label component 
index file need not be regenerated until image registra- 
tion, deletion, or change need be reflected to the label 
component index. Note that, the label component index 
may be added or updated when an image is registered. 
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In step S21, a similar image retrieval specified im- 
age is designated from a user interface unit 11. in step 
S22a. the image ID of the designated specified image 
is acquired, and the label matrix (color label matrix in 
this example) of the specified image is acquired from 
the Image management DB 18a. 

In step S23a, a label matrix group (label matrixes In 
the label system index) including a predetermined 
number or more labels having small penalties to the la- 
bel matrix of the specified image is acquired with refer- 
ence to the label component index file. As in the first 
embodiment, if the label matrix of the specified image Is 
compared with all label matrixes of registered images, 
the processing speed lowers. To prevent this, similar la- 
bel matrixes (label matrix group including a predeter- 
mined number or more labels similar to the label matrix 
of the specified image) are extracted in advance, and 
then, each of these label matrixes Is compared with the 
label matrix o1 the specified image, thereby improving 
the processing speed. If a low processing speed is per- 
mitted, the label matrix of the specified image may be 
compared with all label matrixes of registered images to 
perform highly accurate retrieval (in this case, step S23a 
is omitted). 

In step S24a, each of the label matrixes acquired in 
step S23a is compared to the label nnatrix of the speci- 
fied image to calculate similarity. The label matrixes are 
sequentially output from the closest one to the label ma- 
trix of the specified image together with their similarities 
as a detection result. 

The method of performing similarity comparison be- 
tween label matrixes (calculation of similarity) will be de- 
scribed. The label matrix acquired in step S23a will be 
referred to as an image to be subjected to similarity com- 
parison or image to be compared hereinafter. 

In the second embodiment as well, the penalty ma- 
trix shown in Fig. 12 is used to compare label matrixes 
and obtain the similarity, as in the first embodiment 

More specifically, to give a small penalty (distance) 
to cells adjacent to each other or a large penalty to cells 
spaced apart from each other in pattern matching be- 
tween labels, the penalty matrix between labels as 
shown in Fig. 12 is used. In step S24a, the label matrixes 
are compared with each other in consideration of the 
penalty matrix. In the second embodiment, two-dimen- 
sional DP matching (to be referred to as two-dimension- 
al DP matching hereinafter) to be described below is 
used. 

Figs. 21 A to 21 C are views for explaining similarity 
calculation according to the second embodiment. The 
label matrix of the specified image, which is acquired in 
step S22a, can have an array (a) in Figs. 21 A to 21 C in 
accordance with the scanning method. Assuming that 
one of the label matrixes extracted in step S23a is an 
image to be compared, the label matrix can have an ar- 
ray (b) in Figs. 21 A to 21 C. 

First, the distance between the label string "abc" of 
the first row of the image to be compared and each of 



the label strings of the first to third rows ("123". "456", 
and "789") of the specified image is obtained by DP 
matching. The row number of a label string in the spec- 
ified image, which has the minimum distance, is stored 

5 at a corresponding position of a similar line matrix ((c) 
in Figs. 21 A to 21 C). It the obtained minimum distance 
is larger than a predetemnined threshold value, it is de- 
termined that the label string is not similar to any rows, 
and " ! " is stored at the corresponding position of the sim- 

10 iiar line matrix. Even when the image angle slightly 
changes in the horizontal direction, a similar row (line) 
can be detected by the above processing because of 
the nature of DP matching. The above processing is re- 
peated for all rows ("def " and "ghi") of the image to be 

IS compared: thereby obtaining a similar line matrix (c) in 
the column direction, as shown in Figs. 21 A to 21 C. 

The matrix (c) in Figs. 21 A to 21 C indicate that no 
row similar to "abc" is present in the specified image, 
the first row of the specified image is similar to "def and 

20 the second row of the specified image is similar to "ghi". 
The similarity between the similar line matrix obtained 
in the above manner and a standard line matrix (row ar- 
ray of the specified image: "123" in this example) is ob- 
tained using DP matching and output as the similarity 

2S between the specified image and the image to be com- 
pared. In DP matching, the label sequence to be com- 
pared Is expanded/contracted (the label string to be 
compared with the label sequence is retained without 
advancing to the next one) and compared such that the 

30 similar distance of the label sequence to be compared 
is minimized, as is known. The position up to which ex- 
pansion (retaining) is enabled may be given by a con- 
straint condition (width of matching window). 

Fig. 22 is a flow chart for explaining the procedure 

35 of similarity calculation employing DP matching in the 
second embodiment. Processing which has been de- 
scribed above with reference to Figs. 21 A to 21 C will be 
described in more detail with reference to the flow chart 
in Fig. 22. 

40 In step SI 01, a variable i representing the row 
nunnber of the image to be compared and a variable j 
representing the row number of the specified image are 
initialized to "1 " so that both variables represent the first 
rows. In step SI 02, the label string of the ith row of the 

45 image to be compared is acquired. In Figs. 21 A to 21 C, 
If i = 1 , ■ abc" is acquired. In step 31 03, the label string 
of the jth row of the specified image Is acquired. In Figs. 
21 A to 21 C, if j = 1, "123" is acquired. 

In step SI 04, the distance between the two label 

so strings obtained in steps 8102 and SI 03 is calculated 
by DP matching using the color cell penalty matrix de- 
scribed in Fig. 1 2. In step 8105, if the minimum distance 
is obtained in association with the ith row in step S104, 
the corresponding row number (j) is stored in the line 

55 matrix element LINE [i]. 

Processing in steps SI 03 to SI 05 is performed for 
all rows of the specified image (steps SI 06 and S107). 
In this way, the number of one row of the specified Im- 
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age, which has the minimum distance from the label 
(^'" string of the ith row of the image to be compared. Is 
stored in a line matrix element LtNE[i]. 

In step SI 08, the line matrix element LINE[i] ob- 
tained by the above processing is compared with a pre- 
determined threshold value (Thresh) . If the line matrix 
element LINE [i] is equal to or larger than the threshold 
value Thresh, the flow advances to step S108, and 
representing that the label string is not similar to any 
rows is stored in the line matrix element LINE[i]. 

Processing in steps S102 to SI 08 is repeated for 
all rows of the image to be compared (steps S110 and 
Sill ) to obtain line matrix elements LINE[1] to LINE[im- 
ax]. These line matrix elements are output as a similar 
line matrix LINE[]. 

In step S113, the standard line matrix [12...imax] 
and the similar line matrix LINE[] are subjected to DP 
matching to calculate the distance between the two ma- 
trixes. Note that the standard line matrix means a matrix 
which starts from "1 ' and increases by one in the column 
direction. 

The penalty used in DP matching between the 
standard line matrix and the similar line matrix will be 
described. Two penalty setting schema are available for 
DP matching between the standard line matrix and the 
similar line matrix in the column direction: a dynamic 
penalty and a static penalty. 

In the dynamic penalty, the penalty between line 
numbers Is dynamically set. The penalty between line 
numbers changes depending on the image. In this em- 
bodiment, the distance between two horizontal (row-di- 
rection) label matrixes of the specified image itself is cal- 
culated, and the penalty between rows is obtained on 
the basis of the distance between the label matrixes. 

Fig. 23 is a flow chart showing the procedure of dy- 
namic penalty value setting according to the second em- 
bodiment. In step SI 21, the variable I Is set at "1", and 
the variable j is set at "2". In step SI 22. the label string 
of the ith row of the specified image is acquired. In step 
3123, the label string of the jth row of the specified im- 
age is acquired. In step SI 24, the label string of the ith 
row of the specified image and that of the jth row are 
subjected to DP matching using the color penalty matrix 
to acquire the distance between the label strings. In step 
SI 25, the DP matching distance obtained in step Si 24 
is stored in the line matrix element LINE[i] [j] as the pen- 
alty between the label string of the ith row of the speci- 
fied image and that of the jth row. and simultaneously, 
stored in the line matrix element LINEU][i] as the penalty 
between the label string of the jth row of the specified 
image and that of the ith row. 

Processing in steps 8123 to SI 25 is repeated until 
the variable j equals jmax in step S126. As a result, the 
penalty value between the label string of the ith row and 
each of the label strings of the (i+1 ) th to jmaxth rows Is 
determined. Processing in steps S1 23 to SI 26 is repeat- 
ed until the variable i equals Imax - 1 in steps SI 28, 
SI 29. and SI 30. As a result, the penalty values deter- 



mined by the above processing are stored in the line 
matrix element LINE[i] [j], excluding the diagonal com- 
ponent at I = j. 

In step SI 31 , the penalty of the diagonal component 

5 of the line matrix element LINE[i]G], which has not been 
determined by the above processing, is detemnined. 
Since i = j at this portion, Le.. the label strings are iden- 
tical, the distance between the label strings is 0, and 
penalty "O" is stored. In step SI 32, the penalty for is 

10 detemnined. As the penalty lor "I", a value larger than 
the largest one of all the penalty values in the line matrix 
element LINE[i][j] to some degree is set. However, if the 
penalty value is made excessively large, ambiguous 
matching is disabled. 

is In the above manner. DP matching in step 811 3 is 
performed using the penalty values between the label 
strings of the specified image, thereby acquiring the sim- 
ilarity between the specified image and the image to be 
compared. 

20 On the other hand, in the static penally, if two labels 
match, penalty "0" is given as the penalty for DP match- 
ing; otherwise, or in case of comparison with "!', penalty 
having a certain value is given. In this case, the penalty 
does not change depending on the specified image. 
2S Processing in step S11 3 is executed using such a static 
penalty, thereby determining the similarity between the 
specified image and the image to be compared. 

The above-described matching processing has the 
following characteristic features. If Figs. 21 A and 21 B 
30 are considerably similar, the similar line matrix is "1 23", 
and the distance between the matrixes is 0. II the similar 
line matrix is "M2'' or "212". it is likely that the image to 
be compared has been shifted to the lower side with re- 
spect to the specified image. If the similar line matrix Is 
35 '2Q\' or "233", it is likely that the image to be compared 
has been shifted to the upper side with respect to the 
specified image. If the similar line matrix is "1 3!" or "! 1 3", 
it is likely that the image to be compared is a reduced 
image of the specified Image. If the image to be com- 
40 pared is an enlarged image of the specified image, it can 
also be detected. 

As described above in step S113, when the similar 
line matrix and the standard line matrix are subjected to 
DP matching, a shift in the vertical direction is effectively 
45 absorbed. For this reason, the difference between the 
specified image and the image to be compared due to 
a shift to the upper or lower side, enlargement, or reduc- 
tion of the image can be effectively absorbed, so satis- 
factory -similarity retrieval can be performed. 
so More specifically, the two-dimensional DP matching 
of this embodiment allows certain ambiguity in se- 
quence of label strings of the label matrix and absorbs 
the influence of the positional shift in image. When the 
object position changes due to the angle difference, or 
55 the position of the object to be extracted changes de- 
pending on the block, the hue of the block Is expected 
to finely change. This difference is absorbed by the 
above-described penalty matrix. As described above. 
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by the synergy o1 two-dimensional DP matching allow- 
ing the annbigulty in the second embodiment and the al- 
lowance of ambiguity in feature amounts by the penalty 
matrix, matching less affected by a shift in the vertical 
and horizontal direction: enlargement, or reduction is re- 
alized. 

Of the dynamic penalty and the static penalty, the 
dynamic penalty is more preferable. For the specified 
image of, e.g., a boundless wheat field, all lines may 
have similar label strings. If the image to be compared 
also has the image of a boundless wheat field, the sim- 
ilar line matrix of this image may be "111 " because the 
first line number "1" is stored at all positions. In this case, 
all lines of the specified image have similar images. For 
this reason, unless the penalty between line numbers is 
very small, matching at a small distance cannot be per- 
formed. However, when the dynamic penalty is used, 
the penalty between line numbers becomes small, so 
high similarity can be obtained. 

On the other hand, when the static penally is used, 
the penalty value between "123" and "111" is large, so 
the similarity is low. 

In step S24a. the above-described similarity calcu- 
lation is performed for all label matrixes acquired in step 
S23a- The obtained similarities are sorted in descending 
order, and the flow advances to step S25a. In step S25a, 
an image search is made from the label system index 
using the label matrix with high similarity as a key, and 
the image ID of the found image is acquired. This 
processing is repeated for label matrixes output in de- 
scending order of similarities to obtain the image ID 
group of similar images, in step S26, the full path file 
name for each image ID of the image ID group is ac- 
quired with reference to the image management DB 1 8a 
and presented to the user. 

As described above, when DP matching is per- 
formed in the horizontal and vertical directions, i.e., two- 
dimensionally, retrieval can be performed even when 
the image angle changes in the horizontal, vertical, or 
oblique direction, or the object moves. In addition, a 
zoom-in image or a macroimage can be retrieved be- 
cause of the time-series expansion/contraction charac- 
teristics of DP matching. 

In the second embodiment, a similar line matrix is 
obtained using label strings corresponding to block ar- 
rays in the horizontal direction. However, by the same 
method as described above, a similar line matrix can be 
obtained using label strings corresponding to block ar- 
rays in the vertical direction. 

As described above, according to the second em- 
bodiment, the feature amount group (group of feature 
amounts obtained by segmenting the feature amount 
space) is represented by one symbol (i.e., a label is gen- 
erated), and the distance based on the similarity be- 
tween labels is given on the basis of the above-de- 
scribed two-dimensional DP matching and penalty ma- 
trix. With this arrangement, the amount of calculation of 
the distance between two image blocks can be largely 



reduced. In addition, since similar feature amounts are 
represented by one label, similar image retrieval can be 
satisfactorily performed. 

When (1) the concept of distance between labels 
s based on the penalty matrix is introduced, and (2) two- 
dimensional DP matching allowing to ambiguously 
move the positions of labels to be compared to realize 
label string comparison such that the total distance is 
minimized (similarity is maximized) is introduced, similar 
10 image retrieval can be performed even when the image 
angle slightly changes, so images having similar atmos- 
pheres can be retrieved. 

Furthermore, according to this embodiment, the 
speed of image retrieval can be further increased by us- 
is ing an index database (label system index or label com- 
ponent index). 

As described above, according to each of the above 
embodiments, similar image retrieval can be performed 
at a high speed In consideration of the arrangement of 
20 the feature amounts of images. Simultaneously, similar 
image retrieval can be performed while absorbing dif- 
ferences due to variation in photographing condition. 
Some difference in feature amount between images due 
to a change in image angle, object position, or photo- 
ns graphing condition can be absorbed, unlike the prior art, 
so robust similar image retrieval can be performed. 

As has been described above, according to the 
present invention, high-speed similar image retrieval 
can be performed in consideration of the arrangement 
30 of feature amounts of images. In addition, according to 
the present invention, similar image retrieval can be per- 
formed while effectively absorbing a change in image 
due to a variation in photographing condition such as a 
photographing angle or photographing magnification- 
's In each of the above embodiments, natural image 
retrieval has been described. However, it is obvious to 
a person skilled in the art that the present invention can 
be applied to retrieval of an artificial image of CG or 
CAD. 

40 In each of the above embodiment, color information 
is selected as an image feature amount. However, the 
present invention is not limited to this and can be prac- 
ticed by obtaining another image parameter in units of 
image segmented blocks. 

45 In each of the above embodiments, recognition is 
made on the basis of one feature amount. However, 
high-speed retrieval based on a plurality of feature 
amounts can also be performed by Boolean operations 
of retrieval results based on various feature amounts. 

50 By designating the degree of ambiguity to change 
the width of a so-called matching window in DP match- 
ing, a desired degree of ambiguity for retrieval can be 
set. 

Fig. 24 explains adjustment of width of matching 
55 window of DP matching. As shown in Fig. 24, lines A 
and B define width of the matching window. The line A 
is defined by the equation i=j+r, the line B is defined by 
the wquation i=]-r. The width of the matching window can 



10 

NSDOCID: <EP_0866409A1J_> 



19 



EP 0 866 409 A1 



20 



be determined by value r. Therefore, a desired width of 
f ^ the matching window can be set by designating the val- 
ue r from the keyboard 1 04. Note that, the value r for the 
line A and the value r for the line B may be different from 
each other. 

Furthermore, according to the above embodiments, 
the label component index and the label component in- 
dex are utilized. However, it is apparent that retrieval of 
similar images can be achieced wihtout utilizing such 
indexes. 

When retrieval is to be performed for one image us- 
ing a plurality of image feature amounts, the similarity 
obtained in the present invention may be regarded as a 
new feature amount, and retrieval using statistical dis- 
tance may be performed by multivariate analysis using 
a plurality of parameters. In each of the above embodi- 
ments, a similar image having a similarity larger than a 
predetermined value is obtained as a retrieval result. 
However, a predetermined number of images may be 
sequentially output from an image with high similarity as 
retrieval results, as a matter of course. 

The present invention may be applied to a system 
constituted by aplurality of devices, e.g., a host compu- 
ter, an interface device, a reader, or a printer, or an ap- 
paratus comprising a single device (e.g., a copying ma- 
chine, a facsimile apparatus, or the like) 

The object of the present invention is realized even 
by supplying a storage medium storing software pro- 
gram codes for realizing the functions of the above-de- 
scribed embodiments to a system or an apparatus, and 
causing the computer (or a CPU or an MPU) of the sys- 
tem or apparatus to read out and execute the program 
codes stored in the storage medium. 

In this case, the program codes read out from the 
storage medium realize the functions of the above-de- 
scribed embodiments by themselves, and the storage 
medium storing the program codes constitutes the 
present invention. 

As a storage medium for supplying the program 
codes, a floppy disk, a hard disk, an optical disk, a mag- 
netooptical disk, a CD-ROM, a CD-R, a magnetic tape, 
a nonvolatile memory card, a ROM, or the like can be 
used. 

The functions of the above-described embodiments 
are realized not only when the readout program codes 
are executed by the computer but also when the OS 
(Operating System) running on the computer performs 
part or all of actual processing on the basis of the in- 
structions of the program codes, as a matter of course. 

The functions of the above-described embodiments 
are also realized when the program codes read out from 
the storage medium are written in the memory of a func- 
tion expansion board inserted into the computer or a 
function expansion unit connected to the computer, and 
the CPU of the function expansion board or function ex- 
pansion unit performs part or all of actual processing on 
the basis of the instructions of the program codes, as a 
matter of course. 



As many apparently widely different embodiments 
of the present invention can be made without departing 
from the scope thereof, it is to be understood that the 
invention is not limited to the specific embodiments 
thereof except as defined in the appended claims. 



Claims 

10 1 . An image retrieval apparatus characterized by com- 
prising: 

addition means (14. S11 - S13) for segmenting 
an image into a plurality of blocks and adding 

IS a label to a feature amount acquired in corre- 

spondence with each block; 
generation means (15, SI 4) for arranging la- 
bels added by said addition means on the basis 
of a predetermined block sequence to generate 

20 a label string; 

storage means (18. 18a, 815, S16) for storing 
the label string generated by said generation 
means in correspondence with the image; and 
retrieval means (16, 16a, 822 - 825) for per- 

25 forming image retrieval on the basis of the label 

string stored in said storage means. 

2. The apparatus according to claim 1 , wherein 

30 the label is unique to each cell obtained by seg- 

menting a multidimensional feature amount 
space into a plurality of cells, and 
said addition means calculates the feature 
amount of each block and adds a label of a cell 

35 to which the calculated feature amount belongs 

to the block. 

3. The apparatus according to claim 1 , wherein 

40 the plurality of blocks are obtained by segment- 

ing the image into a plurality of blocks in hori- 
zontal and vertical directions, and the block se- 
quence used by said generation means defines 
an order of obliquely scanning the plurality of 

45 blocks. 

4. The apparatus according to claim 1 , wherein 

the plurality of blocks are obtained by segment- 
so ing the image Into a plurality of blocks in hori- 
zontal and vertical directions, and the block se- 
quence used by said generation means defines 
an order of scanning the plurality of blocks in 
the horizontal or vertical direction. 

55 

5. The apparatus according to claim 1 , wherein 

said retrieval means comprises 
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arithmetic means (16. S22 -S 24) for calculating 
a similarity between a label string of a specified 
image and a label string of an image to be com- 
pared stored in said storage means, and 
output means (1 6, S24 - S26) tor outputting, as 
a retrieval result, an image to be compared 
whose similarity calculated by said arithmetic 
means exceeds a predetermined value. 



a first table for registering an image ID group 
having label strings generated by said genera- 
tion means as keys, 

arithmetic means (16, S23) for calculating a 
similarity between a label string of a specified 
image and a label string of an image to be com- 
pared stored in said storage means, and 
extraction means (S24, S25) for acquiring a la- 
bel string whose similarity calculated by said 
arithmetic means exceeds a predetermined 
value and extracting an image corresponding 
to the acquired label string with relerence to the 
first table. 



11. The apparatus according to claim 10, wherein 

the label string subjected to similarity calcula- 
tion by said arithmetic means is a field In the 
s first table. 

12. The apparatus according to claim 10. wherein 

the label string subjected to similarity calcula- 
tion by said arithmetic means is one of label 
strings stored in said storage means, which in- 
cludes a predetermined number or more labels 
similar to those in the label string of the speci- 
fied image. 

13. The apparatus according to claim 1 2, characterized 
by further comprising 

a second table for registering a label string 
group of label strings stored in said storage 
means, which includes labels as keys, and in 
that 

a label string including a predetermined 
number or more labels similar to those included 
in the specified image is acquired with refer- 
ence to the second table and subjected to sim- 
ilarity calculation by said arithmetic means. 

14. The apparatus according to claim 1 , wherein 

the label string represents a two-dimensional 
label matrix, and 
said retrieval means comprises 
first matching means (SI 01 - S1 12) for making 
a label string extracted from a label matrix of a 
retrieval specified image in units of rows corre- 
spond to a label string extracted from a label 
matrix of an image to be compared in units of 
rows by DP matching to obtain a row array of 
the image to be compared, and 
second matching means (8113) for obtaining, 
by DP matching, a similarity between a row ar- 
ray of the retrieval specified image and the row 
array obtained by said first matching means. 

15. The apparatus according to claim 14. wherein 

the label siring extracted in units of rows has 
an array in the horizontal direction of the image. 
so 

16. The apparatus according to claim 14. wherein 

the label string extracted in units of rows has 
an array in the vertical direction of the image. 

55 

17. The apparatus according to claim 14, characterized 
by further comprising 



6. The apparatus according to claim 5. wherein io 

said arithmetic means has a penalty table 
which holds a penalty value for a pair of label 
values, acquires, with reference to the penalty 
table, the penalty value for the pair of label val- 
ues obtained from the label string of the spec- 
ified image and the label string of the image to 
be compared, and calculates the similarity on 
the basis of the acquired penalty value. 

20 

7. The apparatus according to claim 6, wherein 

the label is unique to each cell obtained by seg- 
menting a multidimensional feature amount 
space into a plurality of cells, and 
the penalty value is set on the basis of a dis- 
tance between cells represented by two labels. 

8. The apparatus according to claim 6, wherein 

30 

said arithmetic means further adds a penalty 
value corresponding to an excess or deficiency 
of labels in calculating the similarity between 
the label string of the specified image and the 
label string of the image to be compared. 3S 

9. The apparatus according to claim 8, wherein 

the penalty value corresponding to the excess 
or deficiency of labels is acquired on the basis 40 
of an automaton theory. 

10. The apparatus according to claim 1 , wherein 

said retrieval means comprises 
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output means (S24a, S25a) for outputting, as a 
retrieval result, an image whose similarity ob- 
tained by said second matching means ex- 
ceeds a predetermined value. 

5 

18. The apparatus according to claim 14. wherein 

said first matching means has a table which 
holds a penalty value for a pair of labels, and 
refers to the table in calculating a distance be- io 
tween the label .string of the retrieval specified 
image and the label string of the image to be 
compared using DP matching. 

19. The apparatus according to claim 18, wherein ^5 

the label is unique to each cell obtained by seg- 
menting a multidimensional feature amount 
space into a plurality of cells, and 
the penalty value is set on the basis of a dis- 20 
tance between cells represented by two labels. 

20. The apparatus according to claim 1 4, wherein 

said second matching means has a table which 2S 
holds a penalty value for a pair of row numbers 
of the row array and refers to the table in cal- 
culating the similarity between the row array of 
the retrieval specified image and the row array 
of the image to be compared using DP match- so 
ing. 

21 . The apparatus according to claim 1 4, characterized 
by further comprising 

35 

holding means (SI 21 - SI 32) for detemnining a 
penalty value associated with a pair of rows on 
the basis of the similarity between label strings 
in a row direction of the retrieval specified im- 
age and holding the penalty value, and in that 40 
said second matching means refers to the pen- 
alty value held by said holding means in calcu- 
latingthe similarity between the row array of the 
retrieval specified image and the row array of 
the image to be compared using DP matching. 45 

22. The apparatus according to claim 18, wherein 

said first matching means applies a penalty and 
constraint according to system expansion/con- so 
traction in label comparison in calculating the 
similarity between the label string of the retriev- 
al specified image and the label string stored in 
said storage means. 

ss 

23. The apparatus according to claim 22, wherein 

the penalty and constraint according to system 



expansion/contraction in label comparison are 
acquired on the basis of a DP matching theory. 

24. The apparatus according to claim 18, characterized 
by further comprising 

a first table for registering an image ID group 
having label strings generated by said genera- 
tion means as keys, and In that 
said output means acquires a label string 
whose similarity calculated by said retrieval 
means exceeds a predetermined value, ex- 
tracts an image corresponding to the acquired 
label string with reference to the first table, and 
outputs the image as a retrieval result. 

25. The apparatus according to claim 24, wherein 

the label matrix subjected to similarity calcula- 
tion by said retrieval means is a key In the first 
table. 

26. The apparatus according to claim 24, wherein 

the label matrix subjected to similarity calcula- 
tion by said retrieval means is one of label ma- 
trixes stored in said storage means, which in- 
cludes a predetermined number or more labels 
similar to those in the label matrix of the retriev- 
al specified image. 

27. The apparatus according to claim 26, characterized 
by further comprising 

a second table for registering a label matrix 
group of label matrixes stored in said storage 
means, which includes labels as keys, and In 
that 

a label matrix including a predetermined 
number or more labels similar to those included 
in the specified image is acquired with refer- 
ence to the second table and subjected to sim- 
ilarity calculatbn by said arithmetic means. 

28. An image retrieval method characterized by com- 
prising: 

the addition step (S11 - SI 3) of segmenting an 
image into a plurality of blocks and adding a la- 
bel to a feature amount acquired in correspond- 
ence with each block; 

the generation step (SI 4) of arranging labels 
added in the addition step on the basis of a pre- 
determined block sequence to generate a label 

string; 

the storage step (S15, SI 6) of storing the label 
string generated in the generation step In cor- 
respondence with the image; and 
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the retrieval step (S22 - S25) of performing im- 
age retrieval on the basis of the label string 
stored in the storage step. 

29. The nnethod according to clainn 28, wherein 

the label Is unique to each cell obtained by seg- 
menting a multidimensional feature amount 
space into a plurality of cells, and 
the addition step includes calculating the fea- 
ture amount of each block and adding a label 
of a cell to which the calculated feature amount 
belongs to the block. 

30. The method according to claim 28. wherein 

the plurality of blocks are obtained by segment- 
ing the image into a plurality of blocks in hori- 
zontal and vertical directions, and the block se- 
quence used in the generation step defines an 
order of obliquely scanning the plurality of 
blocks. 

31. The method according to claim 28, wherein 

the plurality of blocks are obtained by segment- 
ing the image into a plurality of blocks in hori- 
zontal and vertical directions, and the block se- 
quence used in the generation step defines an 
order of scanning the plurality of blocks in the 
horizontal or vertical direction. 

32. The method according to claim 28, wherein 

the retrieval step comprises 
the arithmetic step (S22 - S24) of calculating a 
similarity between a label string of a specified 
image and a label string of an Image to be com- 
pared stored in the storage step, and 
the output step (S24 - S26) of outputting. as a 
retrieval result, an image to be compared 
whose similarity calculated in the arithmetic 
step exceeds a predetermined value. 

33. The method according to claim 32. wherein 

the arithmetic step has a penalty table which 
holds a penalty value for a pair of label values, 
and includes acquiring, with reference to the 
penalty table, the penalty value for the pair of 
label values obtained from the label string of the 
specified image and the label string of the im- 
age to be compared, and calculating the simi- 
larity on the basis of the acquired penalty value. 

34. The method according to claim 33, wherein 

the label is unique to each cell obtained by seg- 



menting a multidimensional feature amount 
space into a plurality of cells, and 
the penalty value is set on the basis of a dis- 
tance between cells represented by two labels. 

5 

35. The method according to claim 33, wherein 

the arithmetic step includes further adding a 
penalty value corresponding to an excess or 
10 deficiency of labels in calculating the similarity 

between the label string of the specified image 
and the label string of the image to be com- 
pared. 

IS 36, The method according to claim 35, wherein 

the penalty value corresponding to the excess 
or deficiency of labels is acquired on the basis 
of an automaton theory. 

20 

37. The method according to claim 28, wherein 

the retrieval step comprises 
a first table for registering an image ID group 
2S having label strings generated in the generation 

step as keys, 

the arithmetic step (S23) of calculating a simi- 
larity between a label string of a specified im- 
age and a label string of an image to be com- 

30 pared stored in the storage step, and 

the extraction step (S24, S25) of acquiring a la- 
bel string whose similarity calculated in the 
arithmetic step exceeds a predetermined value 
and extracting an image corresponding to the 

35 acquired label string with reference to the first 

table. 

38. The method according to claim 37, wherein 

40 the label string subjected to similarity calcula- 

tion in the arithmetic step is a key in the first 
table. 

39. The method according to claim 37, wherein 

45 

the label string subjected to similarity calcula- 
tion in the arithmetic step is one of label strings 
stored in the storage step, which includes a pre- 
determined number or more labels similar to 
50 those in the label string of the specified image. 

40. The method according to claim 39. characterized 
by further comprising 

55 a second table for registering a label string 

group of label strings stored in the storage step, 
which includes labels as keys, and in that 
a label string including a predetermined 
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number or more labels similar to those included 
in the specified image is acquired with refer- 
ence to the second table and subjected to sim- 
ilarity calculation in the arithmetic step. 

41. The method according to claim 28. wherein 

the label string represents a two-dimensional 
label matrix, and 
the retrieval step comprises 
the first matching step (SI 01 - S112) of making 
a label string extracted from a label matrix of a 
retrieval specified image in units of rows corre- 
spond to a label string extracted from a label 
matrix of an image to be compared in units of 
rows by DP nnatching to obtain a row array of 
the image to be compared, and 
the second matching step (S11 3) of obtaining, 
by DP matching, a similarity between a row ar- 
ray of the retrieval specified image and the row 
array obtained in the first matching step. 

42. The method according to claim 41 . wherein 

the label string extracted in units of rows has 
an array in the horizontal direction of the image. 

43. The method according to claim 41 . wherein 

the label string extracted in units of rows has 
an array in the vertical direction of the image. 

44. The method according to claim 41 , characterized 
by further comprising 

the output step (S24a. S25a) of outputting, as 
a retrieval result, an image whose similarity ob- 
tained in the second matching step exceeds a 
predetermined value. 

45. The method according to claim 41 . wherein 

the first matching step has a table which holds 
a penalty value for a pair of labels, and includes 
referring to the table in calculating a distance 
between the label string of the retrieval speci- 
fied Image and the label string of the image to 
be compared using DP matching. 

46. The method according to claim 45. wherein 

the label is unique to each cell obtained by seg- 
menting a multidimensional feature amount 
space into a plurality of cells, and 
the penalty value is set on the basis of a dis- 
tance between cells represented by two labels. 

47. The method according to claim 41 . wherein 



the second matching step has a table which 
holds a penalty value for a pair of row numbers 
of the row array and includes referring to the 
table in calculating the similarity between the 
5 row array of the retrieval specified image and 

the row array of the image to be compared us- 
ing DP matching. 

48. The method according to claim 41 , characterized 
10 by further comprising 

the holding step (S121 - SI 32) of determining 
a penalty value associated with a pair of rows 
on the basis of the similarity between label 
IS strings In a row direction of the retrieval speci- 

fied image and holding the penalty value, and 
in that 

the second matching step includes referring to 
the penalty value held In the holding step in cal- 
20 culating the similarity between the row array of 

the retrieval specified image and the row array 
of the image to be compared using DP match- 
ing. 

2S 49. The method according to claim 45, wherein 

the first matching step includes applying a pen- 
alty and constraint according to system expan- 
sion/contraction in label oonnparison in calcu- 
30 lating the similarity between the label string of 

the retrieval specified image and the label 
string stored in the storage step. 

50. The method according to claim 49, wherein 

3S 

the penalty and constraint according to system 
expansion/contraction in label comparison are 
acquired on the basis of a DP matching theory. 

40 51- The method according to claim 45, characterized 
by further comprising 

a first table for registering an image ID group 
having label strings generated in the generation 

45 step as keys, and in that 

the output step includes acquiring a label string 
whose similarity calculated in the retrieval step 
exceeds a predetermined value, extracting an 
image corresponding to the acquired label 

so string with reference to the first table, and out- 

putting the image as a retrieval result. 

52. The method according to claim 51 . wherein 

55 the label matrix subjected to similarity calcula- 

tion in the retrieval step is a key in the first table. 

53. The method according to claim 51 . characterized in 
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that 

the label matrix subjected to similarity calcula- 
tion in the retrieval step is one of label matrixes 
stored in the storage step, which includes a pre- s 
determined number or more labels similar to 
those in the label matrix of the retrieval speci- 
fied image. 

54. The method according to claim 53. characterized 
by further comprising 

a second table for registering a label matrix 
group of label matrixes stored in the storage 
step, which includes labels as keys, and in that 
a label matrix including a predetermined 
number or more labels similar to those included 
in the specified image is acquired with refer- 
ence to the second table and subjected to sim- 
ilarity calculation in the arithmetic step. 

55, A computer readable memory which stores a con- 
trol program lor performing image retrieval, charac- 
terized in that said control program comprises: 



57. The apparatus according to claim 1 , wherein said 
retrieval means utilizes DP-matchIng method. 

further comprising: 

setting means for setting width of a matching 
window of the DP-matching performed by said 
retrieval means, and 

controlling means for controlling ambiguity of 
said retrieval means by changing the width of 
the matching window based on the width set by 
said setting means. 

58. The method according to claim 28, wherein DP- 
matching method is utilized in said retrieval step, 

further comprising: 

a setting step of setting width of a matching win- 
dow of the DP-matching performed in said re- 
trieval step, and 

controlling step of controlling ambiguity of said 
retrieval step by changing the width of the 
matching window based on the width set in said 
setting step. 
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a code for addition processing of segmenting 
an image into a plurality of blocks and adding 
a label to a feature amount acquired in corre- 
spondence with each block; 

a code for generation processing of arranging 30 
labels added by addition processing on the ba- 
sis of a predetermined block sequence to gen- 
erate a label string; 

a code for storage processing of storing the la- 
bel string generated by generation processing 35 
in correspondence with the image; and 
a code for retrieval processing of performing 
image retrieval on the basis of the label string 
stored by storage processing. 

40 

56. The memory according to claim 55, wherein 

the label string represents a two-dimensional 
label matrix, and 

the code for retrieval processing comprises ^ 
a code for first matching processing of making 
a label string extracted from a label matrix of a 
retrieval specified image in units of rows corre- 
spond to a label string extracted from a label 
matrix of an innage to be compared in units of so 
rows by DP matching to obtain a row array of 
the image to be compared, and 
a code for second matching processing of ob- 
taining, by DP matching, a similarity between a 
row array of the retrieval specified image and ss 
the row array obtained by first matching 
processing. 
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